Supporting mobile mixed-reality experiences

Mobile mixed-reality experiences mix physical and digital spaces, enabling participants to simultaneously inhabit a shared environment online and on the streets. These experiences take the form of games, educational applications and new forms of performance and art, and engender new opportunities for interaction, collaboration and play. As mobile mixed-reality experiences move out of the laboratory and into more public settings they raise new challenges concerning how to support these experiences in the wild. This thesis argues that mobile mixed-reality experiences in which artists retain creative control over the content and operation of each experience, particularly those that are deployed as theatrical performances, require dedicated support for content authoring and reactive orchestration tools and paradigms in order to be successfully and robustly operated in public settings. These requirements are examined in detail, drawing on the experience of supporting four publicly toured mobile mixed-reality experiences; Can You See Me Now?, Uncle Roy All Around You, I Like Frank in Adelaide and Savannah, which have provided a platform to practically develop, refine and evaluate new solutions to answer these challenges in the face of presenting the experiences to many thousands of participants over a four year period. This thesis presents two significant supporting frameworks. The ColourMaps system enables designers to author location-based content by directly colouring over maps; providing a simple, familiar and yet highly flexible approach to matching location-triggers to complex physical spaces. It provides support for multiple and specialised content layers, and the ability to configure and manage other aspects of an experience, including filtering inaccurate position data and underpinning orchestration tools. Second, the Orchestration framework supports the day-to-day operation of public experiences; providing dedicated control-room tools for monitoring that reveal the content landscape and historical events, intervention and improvisation techniques for steering and shaping each participant's experience as it unfolds both physically and virtually, and processes to manage a constant flow of participants

Supporting mobile mixed-reality experiences

Mobile mixed-reality experiences mix physical and digital spaces, enabling participants to simultaneously inhabit a shared environment online and on the streets. These experiences take the form of games, educational applications and new forms of performance and art, and engender new opportunities for interaction, collaboration and play. As mobile mixed-reality experiences move out of the laboratory and into more public settings they raise new challenges concerning how to support these experiences in the wild. This thesis argues that mobile mixed-reality experiences in which artists retain creative control over the content and operation of each experience, particularly those that are deployed as theatrical performances, require dedicated support for content authoring and reactive orchestration tools and paradigms in order to be successfully and robustly operated in public settings. These requirements are examined in detail, drawing on the experience of supporting four publicly toured mobile mixed-reality experiences; Can You See Me Now?, Uncle Roy All Around You, I Like Frank in Adelaide and Savannah, which have provided a platform to practically develop, refine and evaluate new solutions to answer these challenges in the face of presenting the experiences to many thousands of participants over a four year period. This thesis presents two significant supporting frameworks. The ColourMaps system enables designers to author location-based content by directly colouring over maps; providing a simple, familiar and yet highly flexible approach to matching location-triggers to complex physical spaces. It provides support for multiple and specialised content layers, and the ability to configure and manage other aspects of an experience, including filtering inaccurate position data and underpinning orchestration tools. Second, the Orchestration framework supports the day-to-day operation of public experiences; providing dedicated control-room tools for monitoring that reveal the content landscape and historical events, intervention and improvisation techniques for steering and shaping each participant's experience as it unfolds both physically and virtually, and processes to manage a constant flow of participants

Computational models of emotion, personality, and social relationships for interactions in games

We present a novel computational model of emotion, personality and social relationships, implemented and evaluated in an existing commercial game (The Elder Scrolls V: Skyrim). We argue that Non Player Characters (NPCs) with such capabilities will accommodate a new experience in playing games and provide evidence supporting this. Applying the ERiSA Framework [1, 2] to the Skyrim Creation Kit, we designed a simple quest and 2 unique NPCs to interact with. When the ERiSA framework is used, players reported significant changes in their social relationship with the two NPCs compared to the baseline. Most importantly, the results further indicate that the models provide a new experience in playing games. In particular, players report enhanced emotional attachment to the NPCs and appear to forge relationships with the NPCs. Finally, the implemented models result in significant changes in the game engagement and the game immersion score

Play SMILE Game with ERiSA: a user study on game companions

This paper describes the evaluation of our fully integrated virtual game companions framework (ERiSA) [4]. We conducted three user studies with different scenarios using two versions of The Smile Game[4] in semi-public and public spaces. In our study, we show that the game companions' personality was successfully perceived by the participants while interacting and playing with the game companions. Topic about the game itself was the most popular topic with total 598 occurrences in our studies. Moreover, facial expressions is the most performed type of attack in the game. Finally, from the large number of video data collected, we aim to automatically learn the interaction rules and additional attack movements

Bread stories: understanding the drivers of bread consumption for digital food customisation

Consumer demand for food that satisfies specific needs rather than generic mass produced food is growing. In response, the food industry is actively investigating techniques for efficient and comprehensive food customisation. Digital approaches to food customisation are starting to emerge, however, the majority is currently limited to the ingredient level thus excluding consumption drivers such as people’s practices and values around food. Using the approach of cultural probes, we identified four distinct narratives around bread consumption: the healthy bread, the fresh bread, the ethical bread, and the exceptional bread. These themes encapsulate drivers of bread consumption, which we argue can inform the design of digital food innovation platforms

Harnessing the digital records of everyday things

We address how, framed by the Internet of Things, digitally-enabled physical objects may acquire rich digital records throughout their lifetimes, and how these might enhance their value, meaning and utility. We reflect on emerging findings from two case studies, one focusing on wargaming miniatures and the other on an augmented guitar, that engage communities of practice in capturing and utilising rich digital records of things. We articulate an agenda for future research in terms of four key themes: How can the digital records of everyday things be captured using both manual and automated approaches? How can these records enhance the embodied use of things in suitably discrete ways? How can people generate diverse stories and accounts from these records? How can we revisit current notions of ownership to reflect a more fluid sense of custodianship? The findings of the studies reveal common emergent themes and preferences of the practicing communities that surround these objects and the above questions, while ongoing participatory and probe studies continue to reveal nuances and evaluate possible approaches

Modalities of expression: Capturing embodied knowledge in cooking

When cooking we negotiate between instructions in recipes and personal preferences to make in-the-moment creative decisions. This process represents moments of creativity that utilise and reveal our embodied knowledge. This paper focuses on the capture of expressions of embodied knowledge by digitally-networked utensils. We present a design process investigating the design of tangible interfaces to capture and communicate embodied knowledge as a proposition for recipe authoring tools for open innovation in food. We reflect upon this process to discuss lessons about the individual nature of embodied knowledge and its expression, and the context of capturing it to make design recommendations

The multimedia challenges raised by pervasive games

Pervasive gaming is a new form of multimedia entertainment that extends the traditional computer gaming experience out into the real world. Through a combination of personal devices, positioning systems and other multimedia sensors, combined with wireless networking, a pervasive game can respond to a player’s movements and context and enable them to communicate with a game server and other players. We review recent examples of pervasive games in order to explain their distinctive characteristics as multimedia applications. We then consider the challenge of scaling pervasive games to include potentially very large numbers of players. We propose a new approach based upon a campaign model in which individuals, local groups and experts draw on a combination of pervasive games, online services and broadcasting to take part in national or even global events. We discuss the challenges that this raises for further researc

Understanding food consumption lifecycles using wearable cameras

Application of design in HCI is a common approach to engendering behavioural change to address important challenges such as sustainability. Encouraging such change requires an understanding of current motivations and behaviours in the domain in question. In this paper, we describe use of wearable cameras to study motivations and behaviours around food consumption by focusing on two contrasting cultures, Malaysia and the UK. Our findings highlight the potential of wearable cameras to enhance knowledge of food consumption practices and identify where and how some digital interventions might be appropriate to change food behaviour. This includes appealing to people’s motivations behind food consumption and capitalising on existing practices such as gifting of food and social meals. We propose a food consumption lifecycle as a framework to understand and design human–food interaction. The use of wearable cameras enabled us to capture a high-level overview of spatially distributed food-related practices and understand food behaviours in greater depth.This work was co-funded by Horizon Digital Economy Research Institute, UK, and Crops for the Future, Malaysia.This is the author accepted manuscript. The final version is available from Springer via http://dx.doi.org/10.1007/s00779-015-0871-